<!DOCTYPE html>
<html lang="zh-CN">
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width,initial-scale=1">
    <meta name="generator" content="VuePress 2.0.0-beta.66">
    <style>
      :root {
        --c-bg: #fff;
      }
      html.dark {
        --c-bg: #22272e;
      }
      html, body {
        background-color: var(--c-bg);
      }
    </style>
    <script>
      const userMode = localStorage.getItem('vuepress-color-scheme');
			const systemDarkMode = window.matchMedia && window.matchMedia('(prefers-color-scheme: dark)').matches;
			if (userMode === 'dark' || (userMode !== 'light' && systemDarkMode)) {
				document.documentElement.classList.toggle('dark', true);
			}
    </script>
    <title>python爬虫 | 你好， VuePress ！</title><meta name="description" content="这是我的第一个 VuePress 站点">
    <link rel="preload" href="/assets/style-8a38cc2d.css" as="style"><link rel="stylesheet" href="/assets/style-8a38cc2d.css">
    <link rel="modulepreload" href="/assets/app-6c74e544.js"><link rel="modulepreload" href="/assets/python爬虫.html-22e08aed.js"><link rel="modulepreload" href="/assets/python爬虫.html-0eaf5c34.js"><link rel="prefetch" href="/assets/index.html-75ca5b8c.js" as="script"><link rel="prefetch" href="/assets/化学.html-bcd92d03.js" as="script"><link rel="prefetch" href="/assets/IDEA.html-ce9ad873.js" as="script"><link rel="prefetch" href="/assets/python调用idm下载.html-615c23bf.js" as="script"><link rel="prefetch" href="/assets/index.html-571c49cc.js" as="script"><link rel="prefetch" href="/assets/VuePress2x搭建.html-910a2c4e.js" as="script"><link rel="prefetch" href="/assets/WindowsTerminal.html-4a75819f.js" as="script"><link rel="prefetch" href="/assets/乱七八糟.html-0ad5c8cc.js" as="script"><link rel="prefetch" href="/assets/国内访问GitHub.html-bd9bc6eb.js" as="script"><link rel="prefetch" href="/assets/虚拟机安装凤凰OS.html-fd472c84.js" as="script"><link rel="prefetch" href="/assets/软件工程过程.html-ad186a25.js" as="script"><link rel="prefetch" href="/assets/Java.html-a9e8e4b2.js" as="script"><link rel="prefetch" href="/assets/latex.html-b66b07c2.js" as="script"><link rel="prefetch" href="/assets/Markdown.html-76abef01.js" as="script"><link rel="prefetch" href="/assets/Markdown_vuepress.html-7a108572.js" as="script"><link rel="prefetch" href="/assets/Numpy.html-742cd1c8.js" as="script"><link rel="prefetch" href="/assets/python.html-80001bc2.js" as="script"><link rel="prefetch" href="/assets/index.html-404afaec.js" as="script"><link rel="prefetch" href="/assets/Redis.html-3c0347e6.js" as="script"><link rel="prefetch" href="/assets/Selenium.html-dd74eb14.js" as="script"><link rel="prefetch" href="/assets/快速幂算法 牛客小白月赛 1-C分元宵.html-6a00d92c.js" as="script"><link rel="prefetch" href="/assets/数据库.html-a9b825b9.js" as="script"><link rel="prefetch" href="/assets/数据结构_学校.html-db54f988.js" as="script"><link rel="prefetch" href="/assets/404.html-f9875e7b.js" as="script"><link rel="prefetch" href="/assets/index.html-f9be7961.js" as="script"><link rel="prefetch" href="/assets/化学.html-55546e87.js" as="script"><link rel="prefetch" href="/assets/IDEA.html-3d6ee1dc.js" as="script"><link rel="prefetch" href="/assets/python调用idm下载.html-862a9ae4.js" as="script"><link rel="prefetch" href="/assets/index.html-71e4b30e.js" as="script"><link rel="prefetch" href="/assets/VuePress2x搭建.html-69ed53cb.js" as="script"><link rel="prefetch" href="/assets/WindowsTerminal.html-d051c0b9.js" as="script"><link rel="prefetch" href="/assets/乱七八糟.html-e6e19f4d.js" as="script"><link rel="prefetch" href="/assets/国内访问GitHub.html-5a1e283c.js" as="script"><link rel="prefetch" href="/assets/虚拟机安装凤凰OS.html-3f19c5ba.js" as="script"><link rel="prefetch" href="/assets/软件工程过程.html-d5033030.js" as="script"><link rel="prefetch" href="/assets/Java.html-07b672b0.js" as="script"><link rel="prefetch" href="/assets/latex.html-139cf75a.js" as="script"><link rel="prefetch" href="/assets/Markdown.html-24c02676.js" as="script"><link rel="prefetch" href="/assets/Markdown_vuepress.html-a9f40501.js" as="script"><link rel="prefetch" href="/assets/Numpy.html-6ea20a5f.js" as="script"><link rel="prefetch" href="/assets/python.html-da27b5e1.js" as="script"><link rel="prefetch" href="/assets/index.html-25c668e6.js" as="script"><link rel="prefetch" href="/assets/Redis.html-732c58b4.js" as="script"><link rel="prefetch" href="/assets/Selenium.html-8840df78.js" as="script"><link rel="prefetch" href="/assets/快速幂算法 牛客小白月赛 1-C分元宵.html-41e7e00f.js" as="script"><link rel="prefetch" href="/assets/数据库.html-e55ff260.js" as="script"><link rel="prefetch" href="/assets/数据结构_学校.html-cf35a9b9.js" as="script"><link rel="prefetch" href="/assets/404.html-ad53950d.js" as="script"><link rel="prefetch" href="/assets/mermaid-9e549946.js" as="script">
  </head>
  <body>
    <div id="app"><!--[--><div class="theme-container"><!--[--><header class="navbar"><div class="toggle-sidebar-button" title="toggle sidebar" aria-expanded="false" role="button" tabindex="0"><div class="icon" aria-hidden="true"><span></span><span></span><span></span></div></div><span><a href="/" class=""><img class="logo" src="/img/hero.png" alt="你好， VuePress ！"><span class="site-name can-hide">你好， VuePress ！</span></a></span><div class="navbar-items-wrapper" style=""><!--[--><!--]--><nav class="navbar-items can-hide"><!--[--><div class="navbar-item"><div class="navbar-dropdown-wrapper"><button class="navbar-dropdown-title" type="button" aria-label="编程"><span class="title">编程</span><span class="arrow down"></span></button><button class="navbar-dropdown-title-mobile" type="button" aria-label="编程"><span class="title">编程</span><span class="right arrow"></span></button><ul style="display:none;" class="navbar-dropdown"><!--[--><li class="navbar-dropdown-item"><a href="/program/Java.md" class="" aria-label="Java"><!--[--><!--]--> Java <!--[--><!--]--></a></li><li class="navbar-dropdown-item"><a href="/program/Python.md" class="" aria-label="Python"><!--[--><!--]--> Python <!--[--><!--]--></a></li><li class="navbar-dropdown-item"><!--[--><h4 class="navbar-dropdown-subtitle"><span>笔记</span></h4><ul class="navbar-dropdown-subitem-wrapper"><!--[--><li class="navbar-dropdown-subitem"><a href="/program/Markdown.html" class="" aria-label="Markdown"><!--[--><!--]--> Markdown <!--[--><!--]--></a></li><li class="navbar-dropdown-subitem"><a href="/program/latex.html" class="" aria-label="Latex"><!--[--><!--]--> Latex <!--[--><!--]--></a></li><!--]--></ul><!--]--></li><li class="navbar-dropdown-item"><a href="/program/数据库.md" class="" aria-label="数据库"><!--[--><!--]--> 数据库 <!--[--><!--]--></a></li><!--]--></ul></div></div><div class="navbar-item"><a href="/hx/化学.md" class="" aria-label="化学"><!--[--><!--]--> 化学 <!--[--><!--]--></a></div><div class="navbar-item"><a href="/others/" class="" aria-label="乱七八糟"><!--[--><!--]--> 乱七八糟 <!--[--><!--]--></a></div><div class="navbar-item"><a class="external-link" href="https://gitee.com/Francis-xsc/francis-xsc" rel="noopener noreferrer" target="_blank" aria-label="Gitee"><!--[--><!--]--> Gitee <span><svg class="external-link-icon" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" focusable="false" x="0px" y="0px" viewBox="0 0 100 100" width="15" height="15"><path fill="currentColor" d="M18.8,85.1h56l0,0c2.2,0,4-1.8,4-4v-32h-8v28h-48v-48h28v-8h-32l0,0c-2.2,0-4,1.8-4,4v56C14.8,83.3,16.6,85.1,18.8,85.1z"></path><polygon fill="currentColor" points="45.7,48.7 51.3,54.3 77.2,28.5 77.2,37.2 85.2,37.2 85.2,14.9 62.8,14.9 62.8,22.9 71.5,22.9"></polygon></svg><span class="external-link-icon-sr-only">open in new window</span></span><!--[--><!--]--></a></div><!--]--></nav><!--[--><!--]--><button class="toggle-color-mode-button" title="toggle color mode"><svg style="" class="icon" focusable="false" viewBox="0 0 32 32"><path d="M16 12.005a4 4 0 1 1-4 4a4.005 4.005 0 0 1 4-4m0-2a6 6 0 1 0 6 6a6 6 0 0 0-6-6z" fill="currentColor"></path><path d="M5.394 6.813l1.414-1.415l3.506 3.506L8.9 10.318z" fill="currentColor"></path><path d="M2 15.005h5v2H2z" fill="currentColor"></path><path d="M5.394 25.197L8.9 21.691l1.414 1.415l-3.506 3.505z" fill="currentColor"></path><path d="M15 25.005h2v5h-2z" fill="currentColor"></path><path d="M21.687 23.106l1.414-1.415l3.506 3.506l-1.414 1.414z" fill="currentColor"></path><path d="M25 15.005h5v2h-5z" fill="currentColor"></path><path d="M21.687 8.904l3.506-3.506l1.414 1.415l-3.506 3.505z" fill="currentColor"></path><path d="M15 2.005h2v5h-2z" fill="currentColor"></path></svg><svg style="display:none;" class="icon" focusable="false" viewBox="0 0 32 32"><path d="M13.502 5.414a15.075 15.075 0 0 0 11.594 18.194a11.113 11.113 0 0 1-7.975 3.39c-.138 0-.278.005-.418 0a11.094 11.094 0 0 1-3.2-21.584M14.98 3a1.002 1.002 0 0 0-.175.016a13.096 13.096 0 0 0 1.825 25.981c.164.006.328 0 .49 0a13.072 13.072 0 0 0 10.703-5.555a1.01 1.01 0 0 0-.783-1.565A13.08 13.08 0 0 1 15.89 4.38A1.015 1.015 0 0 0 14.98 3z" fill="currentColor"></path></svg></button><!----></div></header><!--]--><div class="sidebar-mask"></div><!--[--><aside class="sidebar"><nav class="navbar-items"><!--[--><div class="navbar-item"><div class="navbar-dropdown-wrapper"><button class="navbar-dropdown-title" type="button" aria-label="编程"><span class="title">编程</span><span class="arrow down"></span></button><button class="navbar-dropdown-title-mobile" type="button" aria-label="编程"><span class="title">编程</span><span class="right arrow"></span></button><ul style="display:none;" class="navbar-dropdown"><!--[--><li class="navbar-dropdown-item"><a href="/program/Java.md" class="" aria-label="Java"><!--[--><!--]--> Java <!--[--><!--]--></a></li><li class="navbar-dropdown-item"><a href="/program/Python.md" class="" aria-label="Python"><!--[--><!--]--> Python <!--[--><!--]--></a></li><li class="navbar-dropdown-item"><!--[--><h4 class="navbar-dropdown-subtitle"><span>笔记</span></h4><ul class="navbar-dropdown-subitem-wrapper"><!--[--><li class="navbar-dropdown-subitem"><a href="/program/Markdown.html" class="" aria-label="Markdown"><!--[--><!--]--> Markdown <!--[--><!--]--></a></li><li class="navbar-dropdown-subitem"><a href="/program/latex.html" class="" aria-label="Latex"><!--[--><!--]--> Latex <!--[--><!--]--></a></li><!--]--></ul><!--]--></li><li class="navbar-dropdown-item"><a href="/program/数据库.md" class="" aria-label="数据库"><!--[--><!--]--> 数据库 <!--[--><!--]--></a></li><!--]--></ul></div></div><div class="navbar-item"><a href="/hx/化学.md" class="" aria-label="化学"><!--[--><!--]--> 化学 <!--[--><!--]--></a></div><div class="navbar-item"><a href="/others/" class="" aria-label="乱七八糟"><!--[--><!--]--> 乱七八糟 <!--[--><!--]--></a></div><div class="navbar-item"><a class="external-link" href="https://gitee.com/Francis-xsc/francis-xsc" rel="noopener noreferrer" target="_blank" aria-label="Gitee"><!--[--><!--]--> Gitee <span><svg class="external-link-icon" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" focusable="false" x="0px" y="0px" viewBox="0 0 100 100" width="15" height="15"><path fill="currentColor" d="M18.8,85.1h56l0,0c2.2,0,4-1.8,4-4v-32h-8v28h-48v-48h28v-8h-32l0,0c-2.2,0-4,1.8-4,4v56C14.8,83.3,16.6,85.1,18.8,85.1z"></path><polygon fill="currentColor" points="45.7,48.7 51.3,54.3 77.2,28.5 77.2,37.2 85.2,37.2 85.2,14.9 62.8,14.9 62.8,22.9 71.5,22.9"></polygon></svg><span class="external-link-icon-sr-only">open in new window</span></span><!--[--><!--]--></a></div><!--]--></nav><!--[--><!--]--><ul class="sidebar-items"><!--[--><li><a href="/program/Java.md" class="sidebar-item sidebar-heading collapsible" aria-label="Java"><!--[--><!--]--> Java <!--[--><!--]--></a><!----></li><li><p tabindex="0" class="sidebar-item sidebar-heading collapsible">Python <span class="right arrow"></span></p><ul style="display:none;" class="sidebar-item-children"><!--[--><li><a href="/program/python.html" class="sidebar-item" aria-label="python-北理工"><!--[--><!--]--> python-北理工 <!--[--><!--]--></a><!----></li><!--]--></ul></li><li><p tabindex="0" class="sidebar-item sidebar-heading collapsible">Markdown <span class="right arrow"></span></p><ul style="display:none;" class="sidebar-item-children"><!--[--><li><a href="/program/Markdown.html" class="sidebar-item" aria-label="Markdown"><!--[--><!--]--> Markdown <!--[--><!--]--></a><!----></li><!--]--></ul></li><!--]--></ul><!--[--><!--]--></aside><!--]--><!--[--><main class="page"><!--[--><!--]--><div class="theme-default-content"><!--[--><!--]--><div><h1 id="python爬虫" tabindex="-1"><a class="header-anchor" href="#python爬虫" aria-hidden="true">#</a> python爬虫</h1><h2 id="第一章-爬虫基础简介" tabindex="-1"><a class="header-anchor" href="#第一章-爬虫基础简介" aria-hidden="true">#</a> 第一章：爬虫基础简介</h2><h2 id="第二章-requests模块基础" tabindex="-1"><a class="header-anchor" href="#第二章-requests模块基础" aria-hidden="true">#</a> 第二章：requests模块基础</h2><ul><li>requests模块、 <ul><li>urllib模块 <ul><li>requests模块</li></ul></li></ul></li></ul><p>requests模块：python中原生的一款基于网络请求的模块，功能非常强大，简单便捷，效率极高。 作用：模拟浏览器发请求。</p><p>如何使用：（requests模块的编码流程）</p><ul><li>指定url <ul><li>UA伪装</li><li>请求参数的处理</li></ul></li><li>发起请求</li><li>获取响应数据</li><li>持久化存储</li></ul><p>环境安装： pip install requests</p><h3 id="_01-requests第一血" tabindex="-1"><a class="header-anchor" href="#_01-requests第一血" aria-hidden="true">#</a> 01.requests第一血</h3><div class="language-python line-numbers-mode" data-ext="py"><pre class="language-python"><code><span class="token comment">#!/usr/bin/env python </span>
<span class="token comment"># -*- coding:utf-8 -*-</span>
<span class="token comment">#- 需求：爬取搜狗首页的页面数据</span>
<span class="token keyword">import</span> requests
<span class="token keyword">if</span> __name__ <span class="token operator">==</span> <span class="token string">&quot;__main__&quot;</span><span class="token punctuation">:</span>
    <span class="token comment">#step_1:指定url</span>
    url <span class="token operator">=</span> <span class="token string">&#39;https://www.sogou.com/&#39;</span>
    <span class="token comment">#step_2:发起请求</span>
    <span class="token comment">#get方法会返回一个响应对象</span>
    response <span class="token operator">=</span> requests<span class="token punctuation">.</span>get<span class="token punctuation">(</span>url<span class="token operator">=</span>url<span class="token punctuation">)</span>
    <span class="token comment">#step_3:获取响应数据.text返回的是字符串形式的响应数据</span>
    page_text <span class="token operator">=</span> response<span class="token punctuation">.</span>text
    <span class="token keyword">print</span><span class="token punctuation">(</span>page_text<span class="token punctuation">)</span>
    <span class="token comment">#step_4:持久化存储</span>
    <span class="token keyword">with</span> <span class="token builtin">open</span><span class="token punctuation">(</span><span class="token string">&#39;./sogou.html&#39;</span><span class="token punctuation">,</span><span class="token string">&#39;w&#39;</span><span class="token punctuation">,</span>encoding<span class="token operator">=</span><span class="token string">&#39;utf-8&#39;</span><span class="token punctuation">)</span> <span class="token keyword">as</span> fp<span class="token punctuation">:</span>
        fp<span class="token punctuation">.</span>write<span class="token punctuation">(</span>page_text<span class="token punctuation">)</span>
    <span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">&#39;爬取数据结束！！！&#39;</span><span class="token punctuation">)</span>

</code></pre><div class="line-numbers" aria-hidden="true"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><h3 id="_02-requests实战之网页采集器" tabindex="-1"><a class="header-anchor" href="#_02-requests实战之网页采集器" aria-hidden="true">#</a> 02.requests实战之网页采集器</h3><div class="language-python line-numbers-mode" data-ext="py"><pre class="language-python"><code><span class="token comment">#!/usr/bin/env python </span>
<span class="token comment"># -*- coding:utf-8 -*-</span>

<span class="token comment">#UA：User-Agent（请求载体的身份标识）</span>
<span class="token comment">#UA检测：门户网站的服务器会检测对应请求的载体身份标识，如果检测到请求的载体身份标识为某一款浏览器，</span>
<span class="token comment">#说明该请求是一个正常的请求。但是，如果检测到请求的载体身份标识不是基于某一款浏览器的，则表示该请求</span>
<span class="token comment">#为不正常的请求（爬虫），则服务器端就很有可能拒绝该次请求。</span>

<span class="token comment">#UA伪装：让爬虫对应的请求载体身份标识伪装成某一款浏览器</span>
<span class="token keyword">import</span> requests
<span class="token keyword">if</span> __name__ <span class="token operator">==</span> <span class="token string">&quot;__main__&quot;</span><span class="token punctuation">:</span>
    <span class="token comment">#UA伪装：将对应的User-Agent封装到一个字典中</span>
    headers <span class="token operator">=</span> <span class="token punctuation">{</span>
        <span class="token string">&#39;User-Agent&#39;</span><span class="token punctuation">:</span><span class="token string">&#39;Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36&#39;</span>
    <span class="token punctuation">}</span>
    url <span class="token operator">=</span> <span class="token string">&#39;https://www.sogou.com/web&#39;</span>
    <span class="token comment">#处理url携带的参数：封装到字典中</span>
    kw <span class="token operator">=</span> <span class="token builtin">input</span><span class="token punctuation">(</span><span class="token string">&#39;enter a word:&#39;</span><span class="token punctuation">)</span>
    param <span class="token operator">=</span> <span class="token punctuation">{</span>
        <span class="token string">&#39;query&#39;</span><span class="token punctuation">:</span>kw
    <span class="token punctuation">}</span>
    <span class="token comment">#对指定的url发起的请求对应的url是携带参数的，并且请求过程中处理了参数</span>
    response <span class="token operator">=</span> requests<span class="token punctuation">.</span>get<span class="token punctuation">(</span>url<span class="token operator">=</span>url<span class="token punctuation">,</span>params<span class="token operator">=</span>param<span class="token punctuation">,</span>headers<span class="token operator">=</span>headers<span class="token punctuation">)</span>

    page_text <span class="token operator">=</span> response<span class="token punctuation">.</span>text
    fileName <span class="token operator">=</span> kw<span class="token operator">+</span><span class="token string">&#39;.html&#39;</span>
    <span class="token keyword">with</span> <span class="token builtin">open</span><span class="token punctuation">(</span>fileName<span class="token punctuation">,</span><span class="token string">&#39;w&#39;</span><span class="token punctuation">,</span>encoding<span class="token operator">=</span><span class="token string">&#39;utf-8&#39;</span><span class="token punctuation">)</span> <span class="token keyword">as</span> fp<span class="token punctuation">:</span>
        fp<span class="token punctuation">.</span>write<span class="token punctuation">(</span>page_text<span class="token punctuation">)</span>
    <span class="token keyword">print</span><span class="token punctuation">(</span>fileName<span class="token punctuation">,</span><span class="token string">&#39;保存成功！！！&#39;</span><span class="token punctuation">)</span>

</code></pre><div class="line-numbers" aria-hidden="true"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><h3 id="_03-requests实战之破解百度翻译" tabindex="-1"><a class="header-anchor" href="#_03-requests实战之破解百度翻译" aria-hidden="true">#</a> 03.requests实战之破解百度翻译</h3><div class="language-python line-numbers-mode" data-ext="py"><pre class="language-python"><code><span class="token comment">#!/usr/bin/env python </span>
<span class="token comment"># -*- coding:utf-8 -*-</span>
<span class="token keyword">import</span> requests
<span class="token keyword">import</span> json
<span class="token keyword">if</span> __name__ <span class="token operator">==</span> <span class="token string">&quot;__main__&quot;</span><span class="token punctuation">:</span>
    <span class="token comment">#1.指定url</span>
    post_url <span class="token operator">=</span> <span class="token string">&#39;https://fanyi.baidu.com/sug&#39;</span>
    <span class="token comment">#2.进行UA伪装</span>
    headers <span class="token operator">=</span> <span class="token punctuation">{</span>
        <span class="token string">&#39;User-Agent&#39;</span><span class="token punctuation">:</span> <span class="token string">&#39;Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36&#39;</span>

    <span class="token punctuation">}</span>
    <span class="token comment">#3.post请求参数处理（同get请求一致）</span>
    word <span class="token operator">=</span> <span class="token builtin">input</span><span class="token punctuation">(</span><span class="token string">&#39;enter a word:&#39;</span><span class="token punctuation">)</span>
    data <span class="token operator">=</span> <span class="token punctuation">{</span>
        <span class="token string">&#39;kw&#39;</span><span class="token punctuation">:</span>word
    <span class="token punctuation">}</span>
    <span class="token comment">#4.请求发送</span>
    response <span class="token operator">=</span> requests<span class="token punctuation">.</span>post<span class="token punctuation">(</span>url<span class="token operator">=</span>post_url<span class="token punctuation">,</span>data<span class="token operator">=</span>data<span class="token punctuation">,</span>headers<span class="token operator">=</span>headers<span class="token punctuation">)</span>
    <span class="token comment">#5.获取响应数据:json()方法返回的是obj（如果确认响应数据是json类型的，才可以使用json（））</span>
    dic_obj <span class="token operator">=</span> response<span class="token punctuation">.</span>json<span class="token punctuation">(</span><span class="token punctuation">)</span>

    <span class="token comment">#持久化存储</span>
    fileName <span class="token operator">=</span> word<span class="token operator">+</span><span class="token string">&#39;.json&#39;</span>
    fp <span class="token operator">=</span> <span class="token builtin">open</span><span class="token punctuation">(</span>fileName<span class="token punctuation">,</span><span class="token string">&#39;w&#39;</span><span class="token punctuation">,</span>encoding<span class="token operator">=</span><span class="token string">&#39;utf-8&#39;</span><span class="token punctuation">)</span>
    json<span class="token punctuation">.</span>dump<span class="token punctuation">(</span>dic_obj<span class="token punctuation">,</span>fp<span class="token operator">=</span>fp<span class="token punctuation">,</span>ensure_ascii<span class="token operator">=</span><span class="token boolean">False</span><span class="token punctuation">)</span>

    <span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">&#39;over!!!&#39;</span><span class="token punctuation">)</span>



</code></pre><div class="line-numbers" aria-hidden="true"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><h3 id="_04-requests实战之豆瓣电影爬取" tabindex="-1"><a class="header-anchor" href="#_04-requests实战之豆瓣电影爬取" aria-hidden="true">#</a> 04.requests实战之豆瓣电影爬取</h3><div class="language-python line-numbers-mode" data-ext="py"><pre class="language-python"><code><span class="token comment">#!/usr/bin/env python </span>
<span class="token comment"># -*- coding:utf-8 -*-</span>
<span class="token keyword">import</span> requests
<span class="token keyword">import</span> json
<span class="token keyword">if</span> __name__ <span class="token operator">==</span> <span class="token string">&quot;__main__&quot;</span><span class="token punctuation">:</span>
    url <span class="token operator">=</span> <span class="token string">&#39;https://movie.douban.com/j/chart/top_list&#39;</span>
    param <span class="token operator">=</span> <span class="token punctuation">{</span>
        <span class="token string">&#39;type&#39;</span><span class="token punctuation">:</span> <span class="token string">&#39;24&#39;</span><span class="token punctuation">,</span>
        <span class="token string">&#39;interval_id&#39;</span><span class="token punctuation">:</span> <span class="token string">&#39;100:90&#39;</span><span class="token punctuation">,</span>
        <span class="token string">&#39;action&#39;</span><span class="token punctuation">:</span><span class="token string">&#39;&#39;</span><span class="token punctuation">,</span>
        <span class="token string">&#39;start&#39;</span><span class="token punctuation">:</span> <span class="token string">&#39;0&#39;</span><span class="token punctuation">,</span><span class="token comment">#从库中的第几部电影去取</span>
        <span class="token string">&#39;limit&#39;</span><span class="token punctuation">:</span> <span class="token string">&#39;20&#39;</span><span class="token punctuation">,</span><span class="token comment">#一次取出的个数</span>
    <span class="token punctuation">}</span>
    headers <span class="token operator">=</span> <span class="token punctuation">{</span>
        <span class="token string">&#39;User-Agent&#39;</span><span class="token punctuation">:</span> <span class="token string">&#39;Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36&#39;</span>

    <span class="token punctuation">}</span>
    response <span class="token operator">=</span> requests<span class="token punctuation">.</span>get<span class="token punctuation">(</span>url<span class="token operator">=</span>url<span class="token punctuation">,</span>params<span class="token operator">=</span>param<span class="token punctuation">,</span>headers<span class="token operator">=</span>headers<span class="token punctuation">)</span>

    list_data <span class="token operator">=</span> response<span class="token punctuation">.</span>json<span class="token punctuation">(</span><span class="token punctuation">)</span>

    fp <span class="token operator">=</span> <span class="token builtin">open</span><span class="token punctuation">(</span><span class="token string">&#39;./douban.json&#39;</span><span class="token punctuation">,</span><span class="token string">&#39;w&#39;</span><span class="token punctuation">,</span>encoding<span class="token operator">=</span><span class="token string">&#39;utf-8&#39;</span><span class="token punctuation">)</span>
    json<span class="token punctuation">.</span>dump<span class="token punctuation">(</span>list_data<span class="token punctuation">,</span>fp<span class="token operator">=</span>fp<span class="token punctuation">,</span>ensure_ascii<span class="token operator">=</span><span class="token boolean">False</span><span class="token punctuation">)</span>
    <span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">&#39;over!!!&#39;</span><span class="token punctuation">)</span>
</code></pre><div class="line-numbers" aria-hidden="true"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><h3 id="_06-requests实战之药监总局相关数据爬取" tabindex="-1"><a class="header-anchor" href="#_06-requests实战之药监总局相关数据爬取" aria-hidden="true">#</a> 06.requests实战之药监总局相关数据爬取</h3><div class="language-python line-numbers-mode" data-ext="py"><pre class="language-python"><code><span class="token comment">#!/usr/bin/env python</span>
<span class="token comment"># -*- coding:utf-8 -*-</span>

<span class="token keyword">import</span>  requests
<span class="token keyword">import</span> json
<span class="token keyword">if</span> __name__ <span class="token operator">==</span> <span class="token string">&quot;__main__&quot;</span><span class="token punctuation">:</span>
    headers <span class="token operator">=</span> <span class="token punctuation">{</span>
        <span class="token string">&#39;User-Agent&#39;</span><span class="token punctuation">:</span> <span class="token string">&#39;Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36&#39;</span>

    <span class="token punctuation">}</span>
    id_list <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span>  <span class="token comment"># 存储企业的id</span>
    all_data_list <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span>  <span class="token comment"># 存储所有的企业详情数据</span>
    <span class="token comment">#批量获取不同企业的id值</span>
    url <span class="token operator">=</span> <span class="token string">&#39;http://125.35.6.84:81/xk/itownet/portalAction.do?method=getXkzsList&#39;</span>
    <span class="token comment">#参数的封装</span>
    <span class="token keyword">for</span> page <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">,</span><span class="token number">6</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
        page <span class="token operator">=</span> <span class="token builtin">str</span><span class="token punctuation">(</span>page<span class="token punctuation">)</span>
        data <span class="token operator">=</span> <span class="token punctuation">{</span>
            <span class="token string">&#39;on&#39;</span><span class="token punctuation">:</span> <span class="token string">&#39;true&#39;</span><span class="token punctuation">,</span>
            <span class="token string">&#39;page&#39;</span><span class="token punctuation">:</span> page<span class="token punctuation">,</span>
            <span class="token string">&#39;pageSize&#39;</span><span class="token punctuation">:</span> <span class="token string">&#39;15&#39;</span><span class="token punctuation">,</span>
            <span class="token string">&#39;productName&#39;</span><span class="token punctuation">:</span><span class="token string">&#39;&#39;</span><span class="token punctuation">,</span>
            <span class="token string">&#39;conditionType&#39;</span><span class="token punctuation">:</span> <span class="token string">&#39;1&#39;</span><span class="token punctuation">,</span>
            <span class="token string">&#39;applyname&#39;</span><span class="token punctuation">:</span><span class="token string">&#39;&#39;</span><span class="token punctuation">,</span>
            <span class="token string">&#39;applysn&#39;</span><span class="token punctuation">:</span><span class="token string">&#39;&#39;</span><span class="token punctuation">,</span>
        <span class="token punctuation">}</span>
        json_ids <span class="token operator">=</span> requests<span class="token punctuation">.</span>post<span class="token punctuation">(</span>url<span class="token operator">=</span>url<span class="token punctuation">,</span>headers<span class="token operator">=</span>headers<span class="token punctuation">,</span>data<span class="token operator">=</span>data<span class="token punctuation">)</span><span class="token punctuation">.</span>json<span class="token punctuation">(</span><span class="token punctuation">)</span>
        <span class="token keyword">for</span> dic <span class="token keyword">in</span> json_ids<span class="token punctuation">[</span><span class="token string">&#39;list&#39;</span><span class="token punctuation">]</span><span class="token punctuation">:</span>
            id_list<span class="token punctuation">.</span>append<span class="token punctuation">(</span>dic<span class="token punctuation">[</span><span class="token string">&#39;ID&#39;</span><span class="token punctuation">]</span><span class="token punctuation">)</span>

    <span class="token comment">#获取企业详情数据</span>
    post_url <span class="token operator">=</span> <span class="token string">&#39;http://125.35.6.84:81/xk/itownet/portalAction.do?method=getXkzsById&#39;</span>
    <span class="token keyword">for</span> <span class="token builtin">id</span> <span class="token keyword">in</span> id_list<span class="token punctuation">:</span>
        data <span class="token operator">=</span> <span class="token punctuation">{</span>
            <span class="token string">&#39;id&#39;</span><span class="token punctuation">:</span><span class="token builtin">id</span>
        <span class="token punctuation">}</span>
        detail_json <span class="token operator">=</span> requests<span class="token punctuation">.</span>post<span class="token punctuation">(</span>url<span class="token operator">=</span>post_url<span class="token punctuation">,</span>headers<span class="token operator">=</span>headers<span class="token punctuation">,</span>data<span class="token operator">=</span>data<span class="token punctuation">)</span><span class="token punctuation">.</span>json<span class="token punctuation">(</span><span class="token punctuation">)</span>
        <span class="token comment"># print(detail_json,&#39;-------------ending-----------&#39;)</span>
        all_data_list<span class="token punctuation">.</span>append<span class="token punctuation">(</span>detail_json<span class="token punctuation">)</span>

    <span class="token comment">#持久化存储all_data_list</span>
    fp <span class="token operator">=</span> <span class="token builtin">open</span><span class="token punctuation">(</span><span class="token string">&#39;./allData.json&#39;</span><span class="token punctuation">,</span><span class="token string">&#39;w&#39;</span><span class="token punctuation">,</span>encoding<span class="token operator">=</span><span class="token string">&#39;utf-8&#39;</span><span class="token punctuation">)</span>
    json<span class="token punctuation">.</span>dump<span class="token punctuation">(</span>all_data_list<span class="token punctuation">,</span>fp<span class="token operator">=</span>fp<span class="token punctuation">,</span>ensure_ascii<span class="token operator">=</span><span class="token boolean">False</span><span class="token punctuation">)</span>
    <span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">&#39;over!!!&#39;</span><span class="token punctuation">)</span>
<span class="token comment">#药监局新的网站http://scxk.nmpa.gov.cn:81/xk/</span>
</code></pre><div class="line-numbers" aria-hidden="true"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><h2 id="第三章-数据解析" tabindex="-1"><a class="header-anchor" href="#第三章-数据解析" aria-hidden="true">#</a> 第三章：数据解析</h2><p>聚焦爬虫:爬取页面中指定的页面内容。 - 编码流程： - 指定url - 发起请求 - 获取响应数据 - 数据解析 - 持久化存储</p><p>数据解析分类：</p><ul><li>正则</li><li>bs4</li><li>xpath（***）</li></ul><p>数据解析原理概述：</p><ul><li>解析的局部的文本内容都会在标签之间或者标签对应的属性中进行存储</li><li>1.进行指定标签的定位 <ul><li>2.标签或者标签对应的属性中存储的数据值进行提取（解析）</li></ul></li></ul><h3 id="正则解析" tabindex="-1"><a class="header-anchor" href="#正则解析" aria-hidden="true">#</a> 正则解析：</h3><div class="language-text line-numbers-mode" data-ext="text"><pre class="language-text"><code>&lt;div class=&quot;thumb&quot;&gt;

&lt;a href=&quot;/article/121721100&quot; target=&quot;_blank&quot;&gt;
&lt;img src=&quot;//pic.qiushibaike.com/system/pictures/12172/121721100/medium/DNXDX9TZ8SDU6OK2.jpg&quot; alt=&quot;指引我有前进的方向&quot;&gt;
&lt;/a&gt;

&lt;/div&gt;

ex = &#39;&lt;div class=&quot;thumb&quot;&gt;.*?&lt;img src=&quot;(.*?)&quot; alt.*?&lt;/div&gt;&#39;

</code></pre><div class="line-numbers" aria-hidden="true"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><h3 id="bs4进行数据解析" tabindex="-1"><a class="header-anchor" href="#bs4进行数据解析" aria-hidden="true">#</a> bs4进行数据解析</h3><ul><li>数据解析的原理： <ol><li>标签定位</li><li>提取标签、标签属性中存储的数据值</li></ol></li><li>bs4数据解析的原理： <ol><li>实例化一个BeautifulSoup对象，并且将页面源码数据加载到该对象中</li><li>通过调用BeautifulSoup对象中相关的属性或者方法进行标签定位和数据提取</li></ol><ul><li>环境安装： <ul><li>pip install bs4</li><li>pip install lxml</li></ul></li></ul></li><li>如何实例化BeautifulSoup对象： <ul><li>from bs4 import BeautifulSoup</li><li>对象的实例化： <ul><li>1.将本地的html文档中的数据加载到该对象中 fp = open(&#39;./test.html&#39;,&#39;r&#39;,encoding=&#39;utf-8&#39;) soup = BeautifulSoup(fp,&#39;lxml&#39;)</li><li>2.将互联网上获取的页面源码加载到该对象中 page_text = response.text soup = BeatifulSoup(page_text,&#39;lxml&#39;)</li><li>提供的用于数据解析的方法和属性：</li><li>soup.tagName:返回的是文档中第一次出现的tagName对应的标签</li><li>soup.find(): <ul><li>find(&#39;tagName&#39;):等同于soup.tagName</li><li>属性定位： <ul><li>soup.find(&#39;div&#39;,class_/id/attr=&#39;song&#39;)</li></ul></li></ul></li><li>soup.find_all(&#39;tagName&#39;):返回符合要求的所有标签（列表）</li><li>select：</li><li>select(&#39;某种选择器（id，class，标签...选择器）&#39;),返回的是一个列表。</li><li>层级选择器： <ul><li>soup.select(&#39;.tang &gt; ul &gt; li &gt; a&#39;)：&gt;表示的是一个层级</li><li>oup.select(&#39;.tang &gt; ul a&#39;)：空格表示的多个层级</li></ul></li></ul></li><li>获取标签之间的文本数据： <ul><li>soup.a.text/string/get_text()</li><li>text/get_text():可以获取某一个标签中所有的文本内容</li><li>string：只可以获取该标签下面直系的文本内容</li><li>获取标签中属性值：</li><li>soup.a[&#39;href&#39;]</li></ul></li></ul></li></ul><p>xpath解析：最常用且最便捷高效的一种解析方式。通用性。</p><ul><li>xpath解析原理： <ul><li>1.实例化一个etree的对象，且需要将被解析的页面源码数据加载到该对象中。</li><li>2.调用etree对象中的xpath方法结合着xpath表达式实现标签的定位和内容的捕获。</li></ul></li><li>环境的安装： <ul><li>pip install lxml</li></ul></li><li>如何实例化一个etree对象:from lxml import etree <ul><li>1.将本地的html文档中的源码数据加载到etree对象中： etree.parse(filePath)</li><li>2.可以将从互联网上获取的源码数据加载到该对象中 etree.HTML(&#39;page_text&#39;)</li><li>xpath(&#39;xpath表达式&#39;)</li></ul></li><li>xpath表达式: <ul><li>/:表示的是从根节点开始定位。表示的是一个层级。</li><li>//:表示的是多个层级。可以表示从任意位置开始定位。</li><li>属性定位：//div[@class=&#39;song&#39;] tag[@attrName=&quot;attrValue&quot;]</li><li>索引定位：//div[@class=&quot;song&quot;]/p[3] 索引是从1开始的。</li><li>取文本： <ul><li>/text() 获取的是标签中直系的文本内容</li><li>//text() 标签中非直系的文本内容（所有的文本内容）</li></ul></li><li>取属性： /@attrName ==&gt;img/src</li></ul></li></ul></div><!--[--><!--]--></div><footer class="page-meta"><div class="meta-item edit-link"><a class="external-link meta-item-label" href="https://gitee.com/Francis-xsc/francis-xsc/edit/main/program/python爬虫.md" rel="noopener noreferrer" target="_blank" aria-label="Edit this page"><!--[--><!--]--> Edit this page <span><svg class="external-link-icon" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" focusable="false" x="0px" y="0px" viewBox="0 0 100 100" width="15" height="15"><path fill="currentColor" d="M18.8,85.1h56l0,0c2.2,0,4-1.8,4-4v-32h-8v28h-48v-48h28v-8h-32l0,0c-2.2,0-4,1.8-4,4v56C14.8,83.3,16.6,85.1,18.8,85.1z"></path><polygon fill="currentColor" points="45.7,48.7 51.3,54.3 77.2,28.5 77.2,37.2 85.2,37.2 85.2,14.9 62.8,14.9 62.8,22.9 71.5,22.9"></polygon></svg><span class="external-link-icon-sr-only">open in new window</span></span><!--[--><!--]--></a></div><div class="meta-item last-updated"><span class="meta-item-label">最后更新时间: </span><!----></div><div class="meta-item contributors"><span class="meta-item-label">Contributors: </span><span class="meta-item-info"><!--[--><!--[--><span class="contributor" title="email: 920364365@qq.com">Francis-xsc</span><!----><!--]--><!--]--></span></div></footer><!----><!--[--><!--]--></main><!--]--></div><!----><!--]--></div>
    <script type="module" src="/assets/app-6c74e544.js" defer></script>
  </body>
</html>
